The present invention relates to a space rendering method for presenting a virtual space to the user while downloading and decoding large-size compressed image data such as ray space data, a virtual space rendering apparatus, a cache device, and a storage medium.
Many schemes that describe and express a virtual space on the basis of actually sensed images in place of a description based on three-dimensional geometric models have been proposed. Such schemes are called Image Based Rendering (to be abbreviated as IBR hereinafter), and are characterized in that they can express a virtual space with high reality, which cannot be obtained by a scheme based on three-dimensional geometric models. Attempts to describe a virtual space on the basis of a ray space theory as one of IBR schemes have been proposed. See, for example, xe2x80x9cImplementation of Virtual Environment by Mixing CG model and Ray Space Dataxe2x80x9d, IEICE Journal D-11, Vol. J80-D-11 No. 11, pp. 3048-3057, November 1997, or xe2x80x9cMutual Conversion between Hologram and Ray Space Aiming at 3D Integrated Image Communicationxe2x80x9d, 3D Image Conference, and the like.
The ray space theory will be explained below. As shown in FIG. 1, a coordinate system 0-X-Y-Z is defined in a real space. A light ray that passes through a reference plane P (Z=z) perpendicular to the Z-axis is defined by a position (x, y) where the light ray crosses P, and variables xcex8 and xcfx86 that indicate the direction of the light ray. More specifically, a single light ray is uniquely defined by five variables (x, y, z, xcex8, xcfx86). If a function that represents the light intensity of this light ray is defined as f, light ray group data in this space can be expressed by f(x, y, z, xcex8, xcfx86). This five-dimensional space is called a xe2x80x9cray spacexe2x80x9d. Generally, a time variation t is also used in some cases, but is omitted here. If the reference plane P is set at z=0, and disparity information of a light ray in the vertical direction, i.e., the degree of freedom in the xcfx86 direction is omitted, the degree of freedom of the light ray care be regenerated to two dimensions (x, xcex8). This x-xcex8 two-dimensional space is a partial space of the ray space. As shown in FIG. 3, if u=tanxcex8, a light ray (FIG. 2) which passes through a point (X, Z) in the real space is mapped onto a line in the x-u space, said line is given by:
X=x+uxc2x7Zxe2x80x83xe2x80x83(1)
Image sensing by a camera reduces to receiving light rays that pass through the lens focal point of the camera by an image sensing surface, and converting their brightness levels and colors into an image. In other words, a light ray group which passes through one point,. i.e., the focal point position, in the real space is captured as an image in correspondence with the number of pixels. In this, since the degree of freedom in the xcfx86 direction is omitted, and the behavior of a light ray is examined in only the X-Z plane, only pixels on a line segment that intersects a plane orthogonal with respect to the Y-axis need to be considered. In this manner, by sensing an image, light rays that pass through one point can be collected, and data on a single line segment in the x-u space can be captured by a single image sensing.
When this image sensing is done a large number of times by changing the viewpoint position (in this specification, the viewpoint position includes both the position of the viewpoint and the line-of-sight direction unless otherwise specified), light ray groups which pass through a large number of points can be captured. When the real space is sensed using N cameras, as shown in FIG. 4, data on a line given by:
Xn=x+uxc2x7Znxe2x80x83xe2x80x83(2)
can be input in correspondence with a focal point position (Xn, Zn) of the n-th camera (n=1, 2, . . . , N) as shown in FIG. 5. In this way, when an image is sensed from a sufficiently large number of viewpoints, the x-u space can be densely filled with data.
Conversely, an observation image from a new arbitrary viewpoint position can be generated (FIG. 7) from the data of the x-u space (FIG. 6). As shown in FIG. 7, an observation image from a new viewpoint position E(X, Z) indicated by an eye mark can be generated by reading out data on a line given by equation (1) from the x-u space.
Actually sensed image data like the aforementioned ray space data is compressed and stored in an external storage device or the like for each unit (e.g., for each object). Therefore, in order to render such space data in a virtual space, the data must be downloaded onto a main storage device, decoded, and rendered on the main storage device. On the other hand, the user can recognize a given virtual space only after virtual images of all virtual objects to be rendered in that virtual space are displayed. Therefore, when there are a plurality of objects to be rendered, the user cannot recognize such virtual objects until space data of all these objects are downloaded, decoded, and rendered. That is, when the user wants to walk through such virtual space, a rendering apparatus with poor response results due to excessive processing requirement.
This is the first problem of the prior art upon handling space data such as ray space data.
The second problem of the prior art results from the fact that actually sensed image data such as ray space data or the like contain a large volume of data. It is a common practice to store such data at a location separated from an image processing apparatus in the form of a database. For this reason, when the image processing apparatus maps a virtual image in a virtual space, a large volume of space data must be downloaded into the image processing apparatus, in advance. Owing to the huge size of actually sensed image data, the turn around time from when space data is requested until that space data is ready to be rendered in the image processing time is not so short, although the communication speed has improved recently. Under these circumstances, the user must be prevented from being bored during the wait time until actually sensed image data is ready for use in such system that presents a virtual space. That is, during this wait time a billboard image (a single image) with a short download time is displayed instead, although a scene from an arbitrary viewpoint position cannot be obtained.
The third problem of the prior art occurs when a walk-through system which allows the user to freely walk through a virtual space using actually sensed image data such as ray space data has a limited memory size. That is, in order to combat the aforementioned first problem, a technique for segmenting a virtual space into a plurality of subspaces (e.g., in case of a virtual art museum, each exhibition room forms one subspace) can be proposed.
More specifically, when it is detected that the user is about to approach a given exhibition room, only space data of that exhibition room is pre-fetched to shorten the time required for transfer prior to rendering. Furthermore, when the user is about to leave that exhibition room (subspace A), space data for the next subspace (e.g., exhibition room B) must be overwritten and stored on the memory area that stored space data of the previous exhibition room(subspace A). In this manner, virtual subspaces of exhibition rooms can be reproduced in turn in nearly real time even by a relatively small memory size.
This pre-fetch start timing is determined depending on whether or not the viewpoint position of the user approaches a target subspace. However, since the user""s viewpoint position moves using a mouse or the like without any high-precision route guide, the user may often be guided to a wrong route. That is, when it is erroneously detected that the user""s viewpoint position which does not reach a pre-fetch start zone has reached that zone, the system starts pre-fetch. Especially, when the user""s viewpoint position moves near the pre-fetch start zone, such operation error readily occurs. For example, as shown in FIG. 8, when the viewpoint position moves from the exhibition room space into the pre-fetch start zone and returns to the exhibition room space again, if the aforementioned wrong pre-fetch starts, large-size space data for the exhibition room space is purged (deleted), and the system must re-transfer the data of the exhibition room space to xe2x80x9creturnxe2x80x9d to the exhibition room space (since this situation takes place due to a detection error of the viewpoint position, the user has no actual sense of returning), resulting in time-consuming operation.
The present invention is directed to the first and third problems. That is, the first object of the present invention against the first problem is to provide a space rendering method and rendering apparatus, which can implement space rendering with high real-time response when large-size space data stored in a memory such as an external storage device or the like is temporarily downloaded, and a virtual space is rendered on the basis of the downloaded space data.
Also, the second object of the present invention against the first problem is to provide a space rendering method and rendering apparatus, which can implement space rendering with high real-time response, when encoded space data is temporarily decoded to render a virtual space.
The object of the present invention against the third problem, is to provide a space rendering method, apparatus, and cache device, which are free from any wasteful re-transfer of the space data.
In order to achieve the first object of the present invention against the first problem, according to one aspect of the present invention, a space rendering method for rendering a virtual space, comprises the steps of:
storing space data in a first format of a virtual object in the virtual space in a first memory;
downloading space data of a virtual object located within a predetermined range from an arbitrary position in the virtual space from the first memory to a second memory when that space data is not present in the second memory; and
rendering the virtual space on the basis of the downloaded space data in the second memory.
According to this aspect, the space data to be downloaded is limited to that near an arbitrary position, thus maintaining high real-time response.
However, even when the space data to be downloaded is limited to that near an arbitrary position as in this aspect, a certain download time is required. Target space data cannot be rendered before downloading ends. Hence, according to the method of another aspect, space data of a virtual object in the virtual space is pre-stored as space data in a second format different from the first format in the second memory, and
while the space data in the first format is being downloaded from the first memory to the second memory, a virtual space is rendered on the basis of the space data in the second format in the second memory. That is, during downloading, the space data of the second format is used as a substitute image.
If the space data of the second format has a large size, the required memory size increases unnecessarily. Hence, according to the method of claim 3, the space data in the first format is space data based on an actually sensed image, and the space data in the second format is billboard image data.
It is not economical to hold the downloaded space data after it becomes unnecessary. Hence, according to the method of yet another aspect of the present invention, when the current position is separated, not less than a predetermined distance, from the position corresponding to the space data already stored in the second memory, the stored space data is released from the second memory.
According to a preferred aspect of the present invention, the first memory is an external memory, and the second memory is a main memory.
Even billboard image data requires a certain memory size. Hence, according to another aspect, the billboard image data is pre-stored in the first memory, and
prior to rendering of the virtual space, the billboard image data in the first memory is downloaded in advance to the second memory.
According to another preferred aspect of the present invention, a virtual walk-through environment is provided to a user in the virtual space.
According to a further preferred aspect of the present invention, the predetermined range indicates a virtual object located within a predetermined distance range from the indicated position. Since the line-of-sight direction is not in question, the probability that the downloaded space data is truly required space data increases.
It is preferable to segment a virtual space into predetermined units, since the required memory size can be reduced. Hence, in one aspect, the virtual object located within the predetermined range from the indicated position includes a virtual object in a virtual subspace located within the predetermined range from the indicated position.
In order to achieve the second object of the present invention against the first problem, a space rendering method for rendering a virtual space, comprises the steps of:
compressing and storing space data in a first format of a virtual object in the virtual space in an internal memory;
decoding a compressed space data of a virtual object located within a predetermined range from an arbitrary position in the virtual space when no decoded space data is present; and
rendering the virtual space on the basis of the decoded space data.
The first and second objects can also be achieved by an apparatus to which the above-described methods are applied, and a storage medium of said computer program.
On the other hand, in order to achieve the object of the present invention against the third problem, a space rendering method for rendering a virtual space, comprises the steps of:
storing space data of a plurality of virtual spaces in a first memory;
rendering a first virtual space at an arbitrary indicated position, which belongs to a first zone, on the basis of first space data, which has been downloaded from the first memory to a second memory different from the first memory; and
holding the first space data in the second memory and downloading space data of a second zone from the first memory to the second memory when the indicated position has moved into the second zone outside the first zone.
According to this aspect, the indicated position which is located within the second zone outside the first zone means that the indicated position may return to the first zone (first virtual space). However, since the first space data is held in the second memory, even when the indicated position returns to the first zone, the first space data need not be re-downloaded. When the indicated position has returned from the second zone to the first zone, in one aspect the first space data held in the second memory is used as a basis of rendering of the virtual space.
In order to effectively use the second memory, when it is obvious that the first space data is unnecessary, invalidating the first space data is preferable. Hence, according to another aspect, when the indicated position has moved from the first zone to the second zone, and has then moved into a third zone outside the second zone, the first space data held in the second memory is released.
It is preferable to release space data not only in response to user operation but also automatically, since the operability then improves. Hence, according to another preferred aspect of the present invention, when the indicated position has moved from the first zone to the second zone, and then stays in the second zone, a measurement of a staying time in the second zone is started, thus preparing for automatic release. When the indicated position has moved from the first zone to the second zone, and then stays in the second zone for a first time duration, the first space data held in the second memory can be released.
It is also preferable to allow the user to arbitrarily set an automatic release time. Hence, when the indicated position has moved from the first zone to the second zone, and a user makes predetermined instruction operation while the indicated position stays in the second zone, the first space data held in the second memory can be released.
According to a further preferred aspect of the present invention, the third zone bounds both the second zone of the first virtual space and a second zone for the second virtual space. For example, in a walk-through environment in which the user can walk through across a plurality of virtual spaces, it is preferable that a zone which is not strongly linked to any virtual space be present, so as to prevent unwanted downloading or memory release from frequently occurring.
In actual operation, the indicated position may repetitively enter and leave the first and second zones. In such case, repetition of downloading of the second space data is reduced to prevent wasteful downloading of the second space data.
According to still another preferred aspect of the present invention, when the indicated position enters the second zone of the first virtual space from the third zone, it is confirmed if the first space data is already held in the second memory. The geometric nature of the second zone as a transition zone of the indicated position among a plurality of virtual spaces can be effectively used.
The object of the present invention against the third problem can also be achieved by a program storage medium which stores a computer program that implements a space rendering method described above.
The object of the present invention against the third problem can also be achieved by a space rendering apparatus implementing a method described above.
The object of the present invention can also be achieved by a cache device. That is, a cache device for caching space data in a cache memory to allow a user to walk through across a plurality of virtual space, comprises:
a cache memory for storing first space data of a first virtual space to allow the user to walk through the first virtual space; and
cache means for, when the user walks through a peripheral zone of the first virtual space, holding the already stored first space data in said cache memory, and pre-fetching space data of a second virtual space which neighbors the first virtual space from an external memory device into said cache memory.
According to another preferred aspect of the present invention, when the user leaves the peripheral zone, a memory area of the first space data in said cache memory is released.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.